This invention relates to an image processing apparatus, and more particularly to an image processing apparatus suitable for use for production of a color image signal of a wide dynamic range from an image signal acquired, for example, using a CCD image sensor of the single plate type or the like.
A solid-state image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Mental-Oxide Semiconductor) is utilized widely in image pickup apparatus such as a video camera and a digital still camera, part inspection apparatus in the field of the FA (Factory Automation) and optical measuring instruments such as an electronic endoscope in the field of the ME (Medical Electronics).
Conventionally, a method is known wherein light intensity signals measured with different sensitivities among different pixels are synthesized in order to increase the dynamic range of image pickup apparatus and optical measuring instruments in which a solid-state image pickup device is used. In the following, first to fourth related-art methods of the type mentioned are described.
As the first related-art method, a method can be listed wherein incoming light beams branched to a plurality of optical axes having different optical transmission factors are measured by solid-state image pickup devices disposed on the individual optical axes. This method is disclosed in the official gazette of Japanese Patent Laid-Open No. Hei 8-223491 and so forth. However, the first method has a problem in that it is disadvantageous in terms of the reduction of the cost or the reduction of the space because it requires a plurality of solid-state image pickup devices and a complicated optical system for branching light.
As the second related-art method, a method can be listed wherein a single solid-state image pickup device is used such that the exposure time thereof is divided into a plurality of time periods to pick up a plurality of images and then the images are synthesized. This method is disclosed in the official gazette of Japanese Patent Laid-Open No. Hei 8-331461 and so forth. However, the second method has a problem in that an image of a dynamic scene in which the intensity of light varies every moment cannot be picked up properly because the information measured with the different sensitivities are picked up at different points of time and with different time widths.
As the third related-art method, a method can be listed wherein a single solid-state image pickup device is used such that a plurality of light receiving elements adjacent each other on an image pickup face thereof form a set which corresponds to one pixel of an output image and have sensitivities different from each other to pick up an image. This method is disclosed in the official gazette of U.S. Pat. No. 5,789,737. As a method for making the sensitivities of light receiving elements which form a solid-state image pickup device different from each other, a method is available wherein the light receiving elements are covered with filters having transmission factors different from each other. Further, a technique which adapts the third related-art method to a color image is disclosed in the official gazette of Japanese Patent Laid-Open No. 2000-69491.
The third related-art method is advantageous in terms of the reduction of the cost and the reduction of the space in terms of which the first related-art method is disadvantageous. Further, the third related-art method can solve the problem of the second related-art method that an image of a dynamic scene cannot be picked up properly. However, with the third related-art method, since a plurality of light receiving elements adjacent each other form a set and correspond to one pixel of an output image, in order to secure a resolution of output pixels, a number of image pickup devices including a number of light receiving elements equal to several times the number of pixels of the output image, resulting in a subject that a large unit cell size is required.
As the fourth related-art method, a method can be listed wherein an image pickup device having an ordinary dynamic range is used to pick up an image with a mechanism applied thereto which makes the exposure different for each light receiving element corresponding to one pixel of an output image and the resulting image signals are subject to predetermined image processing to produce an image signal of a wide dynamic range. The mechanism for making the exposure different among different light receiving elements is implemented by producing a spatial sensitivity pattern by changing the light transmission factor or the numerical aperture for each light receiving element. This method is disclosed in a document ‘S. K. Nayar and T. Mitsunaga, “High Dynamic Range Imaging: Spatially Varying Pixel Exposures”, Proc. of Computer Vision and Pattern Recognition 2000, Vol. 1, pp. 472-479, June, 2000’.
In the fourth related-art method, each of the light receiving elements has only one kind of sensitivity. Consequently, each of pixels of an image picked up can acquire information of a dynamic range which the image pickup device originally has. However, by applying predetermined image processing to resulting image signals so that the sensitivities of all of the pixels may become equal to one another, an image having a wide dynamic range can be produced. Further, since all of the light receiving elements are exposed to light at the same time, an image of a subject having some movement can be picked up properly. Furthermore, since one light receiving element corresponds to one pixel of output image, the problem that a great unit size is required does not occur with the fourth related-art method.
As described above, the fourth related-art method can solve the problems of the first to third related-art methods. However, the fourth related-art method has a premise that a monochromatic image is produced, and has a subject that a technique for producing a color image has not been established. More particularly, the fourth related-art method has a subject that a technique of producing image signals of all color components for all pixels from an image having different colors and/or different sensitivities among different pixels and making the sensitivity uniform has not conventionally been established.